Apparatus and method for multivariate prediction of contact center metrics using machine learning

ABSTRACT

In a predictor device, a method for predicting a metric of a contact center includes receiving contact center operational data associated with a time duration; training a set of algorithms and their available hyperparameters with the contact center operational data to generate a set of data models; generating a score associated with each data model of the set of data models, the score quantifying a performance of each algorithm and its available hyperparameters on the contact center operational data; identifying the data model having the largest score as a best learning model for the time duration; and generating a contact center metric prediction based on the best learning model for the time duration.

RELATED APPLICATIONS

This patent application claims the benefit of U.S. ProvisionalApplication No. 62/817,221, filed on Mar. 12, 2019, entitled, “Apparatusand Method for Multivariate Prediction of Contact Center Metrics Using,”the contents and teachings of which are hereby incorporated by referencein their entirety.

BACKGROUND

In conventional customer contact centers, incoming communications, suchas voice calls or texts for example, can be received and answered by anagent pool. During operation, the contact center can automaticallydistribute and connect incoming communications to available agents or toagents best suited to handle the communications. In the case where nosuitable agents are available, the customer contact center can becomeoverloaded and can place the communications in a variety of queues basedupon some pre-established criteria, such as based on the order ofarrival and/or priority.

One of the priorities of a contact center is to be predictive of itsload, which allows for better planning and scheduling of the resourcesavailable. Another priority of the contact center is to accurately setexpectations with the customer and deliver on those establishedexpectations once set. Setting and delivering on the establishedexpectations closely relates to customer experience as well as servicelevels.

Both of these priorities can be achieved by applying machine learningalgorithms to a variety of contact center operational data such as agentstaffing, call arrival rate, call handling rate, and seasonality, forexample. Such an approach can provide the ability for the contact centerto provide a metric known as estimated wait time (EWT). The estimatedwait time can identify an amount of time a customer is estimated to waitbefore being serviced by an agent. Based upon application of a varietyof machine learning algorithms and their corresponding hyperparametersto contact center operational data, the contact center EWT can bederived which, in turn, can be used to identify the appropriate numberof agents to be staffed by the contact center, as well as to set theexpectation with the customer by reporting or acting within theestimated wait time. The EWT can also be utilized to identify wherecallers should be routed in the contact center (e.g., by placing thecallers in a queue with the lowest wait time).

SUMMARY

The success of any contact center is based upon the accuracy of the EWTestimation. For example, based upon the EWT estimation, the customercontact center can make decisions to efficiently route calls or todetermine whether or not an offer for a callback should be placed. Theaccuracy of the EWT depends upon the accuracy of the conventionalcontact center's statistics, since traditional statistical rules andheuristics are utilized to produce accurate EWTs. However, the processof producing these EWTs can suffer from a variety of deficiencies.

For example, accuracy of the EWT can decrease when the customer contactcenter is overloaded or unstable. Further, contact center metrics canexperience a high variance, such as those based on seasonality, as wellas unexpected and expected events, such as power outages and holidays.Such variance can influence the contact center's statistics andresulting EWT estimates. Additionally, the statistics related to acontact center's operation can change over time. While a customercontact center may be manually optimized for a given time frame, thecontact centers can evolve and change over time as the customers' needschange. Therefore, an accurate EWT for one time period may not beconsidered accurate for a later period.

By contrast to conventional detection of contact center metrics,embodiments of the present innovation relate to an apparatus and methodfor multivariate prediction of contact center metrics, such as EWT,using machine learning. In one arrangement, a predictor device includesa Learner of Learners engine configured to utilize available contactcenter operational data to derive a best learning model for a given timeperiod or season based upon a variety of machine learning algorithmstrained with available hyperparameters. After having identified a bestlearning model, the predictor device can utilize the best learning modelto predict a given contact center metric, such as estimated wait time,for the contact center for the given time period. By using a Learner ofLearners approach, the predictor device can automatically determine thebest model to predict a contact center metric, such as EWT, based on itsperformance. Further, the predictor device makes the modelling processefficient and adaptive, thereby mitigating or eliminating manual stepsand heuristics during operation while adapting to the variance andinstability typically found in conventional contact centers.

In one arrangement, the predictor device also includes a Predictor ofPredictors engine configured to account for short-term fluctuations inthe contact center operational data received from the contact center.The Predictor of Predictors engine is configured to apply updatedcontact center operational data to the previous best learning modelsgenerated by the Learner of Learners engine. Further, the Predictor ofPredictors engine can weigh the previous best learning models whichallows the Predictor of Predictors engine to identify the best model touse when predicting contact center metrics (e.g., estimated wait times).As such, the Predictor of Predictors engine can account for seasonalitywhile applying multivariate prediction to the updated contact centeroperational data based on previously trained hyperparameters andalgorithms.

Further, in one arrangement, the predictor device is configured to applyquality metrics to the models to ensure the use of the most accuratemodel during operation. For example, the predictor device can applytraining quality metrics to a set of data models to identify the bestlearning model. The predictor device can also apply training qualitymetrics to a currently-deployed best learning model to evaluate itsquality during use. In the case where the predictor device identifies adeviation in operation of the best learning model between a predictiondata set and a training data set, the predictor device can be configuredto retrain the best learning model or to select an alternate model asthe best learning model for deployment.

In one arrangement, embodiments of the innovation relate to, in apredictor device, a method for predicting a metric of a contact center.The method includes receiving contact center operational data associatedwith a time duration; training a set of algorithms and their availablehyperparameters with the contact center operational data to generate aset of data models; generating a score associated with each data modelof the set of data models, the score quantifying a performance of eachalgorithm and its available hyperparameters on the contact centeroperational data; identifying the data model having the largest score asa best learning model for the time duration; and generating a contactcenter metric prediction based on the best learning model for the timeduration.

In addition, this method includes continually and automaticallyperforming partial training on the real-time contact center operationaldata in order to keep data models up-to-date while mitigating oreliminating expansive full data training operations.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing and other objects, features and advantages will beapparent from the following description of particular embodiments of theinnovation, as illustrated in the accompanying drawings in which likereference characters refer to the same parts throughout the differentviews. The drawings are not necessarily to scale, emphasis instead beingplaced upon illustrating the principles of various embodiments of theinnovation.

FIG. 1 illustrates a schematic representation of a contact center,according to one arrangement.

FIG. 2 illustrates a schematic representation of a predictor device ofthe contact center of FIG. 1, according to one arrangement.

FIG. 3 illustrates a flowchart showing a process performed by thepredictor device of FIG. 2, according to one arrangement.

FIG. 4 illustrates a schematic representation of a best learnerdatabase, according to one arrangement.

FIG. 5 illustrates a schematic representation of the predictor device ofFIG. 2 having a predictor of predictors engine, according to onearrangement.

FIG. 6 illustrates a schematic representation of the predictor device ofFIG. 2 configured to apply quality metrics to data models, according toone arrangement.

FIG. 7 illustrates a schematic representation of the predictor device ofFIG. 1 configured to evaluate a quality score of the best learningmodel, according to one arrangement.

DETAILED DESCRIPTION

Embodiments of the present innovation relate to an apparatus and methodfor multivariate prediction of contact center metrics, such as EWT,using machine learning. In one arrangement, a predictor device includesa Learner of Learners engine configured to utilize available contactcenter operational data to derive a best learning model for a given timeperiod or season using a variety of machine learning algorithms trainedwith available hyperparameters. After having identified a best learningmodel, the predictor device can utilize the best learning model topredict a given contact center metric, such as estimated wait time, forthe contact center for the given time period. By using a Learner ofLearners approach, the predictor device can automatically determine thebest model to predict a contact center metric, such as EWT, based on itsperformance. Further, the predictor device makes the modelling processefficient and adaptive, thereby mitigating or eliminating manual stepsand heuristics during operation while adapting to the variance andinstability typically found in conventional contact centers.

In one arrangement, the predictor device also includes a Predictor ofPredictors engine configured to account for short-term fluctuations inthe contact center operational data received from the contact center.The Predictor of Predictors engine is configured to apply updatedcontact center operational data to the previous best learning modelsgenerated by the Learner of Learners engine. Further, the Predictor ofPredictors engine can weigh the previous best learning models whichallows the Predictor of Predictors engine to identify the best model touse when predicting contact center metrics (e.g., estimated wait times)for a contact center. As such, the Predictor of Predictors engine canaccount for seasonality while applying multivariate prediction to theupdated contact center operational data based on previously trainedhyperparameters and algorithms.

In one arrangement, the predictor device is configured to apply qualitymetrics to the models to ensure the use of the most accurate modelduring operation. For example, the predictor device can apply trainingquality metrics to a set of data models to identify the best learningmodel. The predictor device can also apply training quality metrics to acurrently-deployed best learning model to evaluate its quality duringuse. In the case where the predictor device identifies a deviation inoperation of the best learning model between a prediction data set and atraining data set, the predictor device is configured to retrain thebest learning model or to select an alternate model as the best learningmodel for deployment.

FIG. 1 illustrates a schematic representation of a contact center 100,according to one arrangement. The contact center 100 can include aserver device 112 disposed in electrical communication with one or moredata stores or databases 114 and a predictor device 116 disposed inelectrical communication with the server device 112.

The server device 112 can be a computerized device having a controller113, such as a processor and memory. According to one arrangement,server device 112 is disposed in electrical communication with a userdevice 118, such as a telephone, smartphone, or tablet device, via anetwork 120, such as a local area network (LAN), a wide area network(WAN), or a public switched telephone network (PSTN). During operation,the server device 112 is configured to direct a user 122 of the userdevice 118, or customer, to an appropriate working agent 124. Eachworking agent 124 can operate a corresponding computer work station 126,such as a personal computer, telephone, tablet device or other type ofvoice communications equipment, all interconnected by a network 128,such as a LAN or WAN 128. Also during operation, the server device 112can store information regarding the user communication to the database114. For example, server device 112 can store contact or customerrelated information for each communication session, as well as otherinformation that can enhance the value and efficiency of the contactinformation.

The predictor device 116 can be a computerized device having acontroller 117, such as a processor and memory. The predictor device 116is configured to identify and self-optimize statistical models basedupon the application of a variety of algorithms and availablehyperparameters to contact center operational data received from theserver device 112, thereby reducing the need for human involvement. Aswill be described below, the predictor device 116 can adjust forseasonality associated with the contact center operational data whilestill training on updated contact center operational data in arelatively fast manner.

As illustrated in FIG. 2, the controller 117, associated contact centeroperations data 136 stored in database 132, etc., are all deployed on asingle computerized device (i.e., the predictor device 116). Suchillustration is by way of example only. In one arrangement, deploymentmodels of the present innovation are flexible and can be deployed andscaled across multiple computerize devices.

With reference to FIG. 2, the predictor device 116 includes a monitoringservice 130, a database 132, and a Learner of Learners (LOL) engine 134.The monitoring service 130 is configured to retrieve contact centeroperational data 136 from the server device 112. For example, duringoperation, the server device 112 can route or direct users 122 toappropriate working agents 124. As such, the contact center operationaldata 136 can relate to information associated with communication routingwithin the contact center 100 for a given period of time. As themonitoring service 130 receives the contact center operational data 136,the monitoring service 130 stores the data 136 to the database 132 foruse by the LOL engine 134.

In one arrangement, the LOL engine 134 can include an algorithm andassociated hyperparameter configuration store 138. The store 138 caninclude an algorithm reference list 139 and a hyperparameter optionslist 141 which can identify any number of algorithms and availablehyperparameters. In one arrangement, the algorithms are configured asregression algorithms. For example, the LOL engine 134 can include, as afirst algorithm in the algorithm reference list 139, a neural networkalgorithm having a variety of available hyperparameters identified inthe hyperparameter options list 141. For example, the neural networkalgorithm can have available hyperparameters related to hidden layersize and available hyperparameters related to learning rate. Further,the LOL engine 134 can include, as a second algorithm in the algorithmreference list 139, a decision tree algorithm having a variety ofavailable hyperparameters identified in the hyperparameter options list141. For example, the decision tree algorithm can have availablehyperparameters related to maximum tree depth. While the store 138 canbe preconfigured with an algorithm reference list 139 and ahyperparameter options list 141, in one arrangement, the store 138 canreceive additional algorithms and hyperparameters from an externalsource and/or have algorithms and available hyperparameters within thelists 139, 141 adjusted or deleted by an external source.

The LOL engine 134 is configured to execute a hyperparameter andalgorithm optimization (HAO) function 140 to utilize the availablecontact center operational data 136 from the database 132 to train a setof algorithms and their available hyperparameters to generate datamodels. As provided below, following the training, the LOL engine 134can determine a best learning model of the data models for a given timeperiod.

FIG. 3 illustrates a flowchart 200 of a process performed by thepredictor device 116 when predicting a metric of a contact center 100,as based upon the determination of a best learning model for a giventime period.

During operation, in step 202, the predictor device 116 receives contactcenter operational data 136 associated with a time duration. Forexample, with reference to FIG. 2, the predictor device 116 provides APIcall instructions 135 to the monitor service 130 which directs themonitor service 130 to retrieve contact center operational data 136 fromthe server device 112 for a particular time period, such as for a givenmonth (e.g., the month of January). The monitor service 130 can storethe retrieved contact center operational data 136 within database 132.

In one arrangement, the predictor device 116 also provides an API call137 to the LOL engine 134. The API call 137 causes the LOL engine 134 toaccess the algorithm and associated hyperparameter configuration store138 and retrieve the algorithms from the reference list 139 andassociated parameters from the options list 141 to be trained on thecontact center operational data 136. The store 138 can include thealgorithm reference list 139 and the hyperparameter options list 141which can identify any number of algorithms and availablehyperparameters. For example, assume the case where the algorithm andassociated hyperparameter configuration store 138 references a linearregression algorithm and a neural network algorithm having two availablehyperparameters: a first hyperparameter identifying the neural networkas having 3 layers with 200 nodes and a second hyperparameteridentifying the neural network as having 2 layers and 50 nodes. In thiscase, the LOL engine 134 can retrieve both algorithms, as well as bothof the hyperparameters associated with the second algorithm.

Further, the API call 137 to the LOL engine 134 can cause the LOL engine134 to access the database 132 to retrieve the contact centeroperational data 136 for a particular time period. For example, inresponse to the API call, the LOL engine 134 can retrieve, as thecontact center operational data 136, communication routing informationwithin the contact center 100 for the month of January.

In one arrangement, the LOL engine 134 is configured to normalize thecontact center operational data 136 retrieved from the database 132. Forexample, the contact center operational data 136 can includecommunication routing information having a variety of time scales. Forexample, the communication routing information can be presented inseconds (s) or milliseconds (ms). In such an example, the LOL engine 134is configured to format the contact center operational data 136 to acommon time scale.

Returning to FIG. 3, at step 204, the predictor device 116 is configuredto train a set of algorithms and their available hyperparameters withthe contact center operational data 136 to generate a set of data models150. In one arrangement, with reference to FIG. 2, the LOL engine 134can execute the hyperparameter and algorithm optimization function 140to apply the contact center operational data 136 to each of thealgorithms and hyperparameters to generate a data model 150 associatedwith each algorithm and hyperparameter case.

For example, assume the store 138 includes an algorithm reference list139 and a hyperparameter options list 141 that identify the followingalgorithms and hyperparameters:

-   -   Algorithm 1: Neural Network    -   Hyperparameters:        -   Hidden Layer Size—[100, 50]        -   Learning Rate—[1,2]    -   Algorithm 2: Decision Tree    -   Hyperparameters:        -   Maximum Tree Depth—[1,2,3]            By applying the contact center operational data 136 to each            of the algorithms and hyperparameters, the LOL engine 134            can generate the following set of data models 150:

Algorithm Hyperparameter Set Model 1 100, 1 Model 1 1  50, 1 Model 2 1100, 2 Model 3 1  50, 2 Model 4 2 1 Model 5 2 2 Model 6 2 3 Model 7

Returning to FIG. 3, at step 206, the predictor device 116 is configuredto generate a score 152 associated with each data model of the set ofdata models 150 where the score 152 quantifies a performance of eachalgorithm and available hyperparameters on the contact centeroperational data 136. Typically, following the generation of a machinelearning model, the model can be quantitatively evaluated using avariety of scoring methods to indicate how well the model performs. Inone arrangement, with reference to FIG. 2, the LOL engine 134 can applya scoring function 154 to each of the data models 150 in order togenerate corresponding scores 152 identifying the performance of each.For example, in the above case, assume the application of the scoringfunction 154 to each of the data models 150 results in the followingscores:

Model Score Model 1 90% Model 2 50% Model 3 70% Model 4 80% Model 5 55%Model 6 60% Model 7 70%

Returning to FIG. 3, at step 208, the predictor device 116 is configuredto identify the data model 150 having the largest score as a bestlearning model 155 for the time duration. For example, based upon aninter-comparison of each score, the predictor device 116 can identifyModel 1 (i.e., neural network with a hidden layer size of 100, and alearning rate of 1) as having the largest score (90%) of the set ofmodels 150 for a given time period, in this case the month of January.As such, the predictor device 116 can identify Model 1 as being the bestlearning model 155 for the set of models 150.

With continued reference to FIG. 3, in step 210, the predictor device116 is configured to generate a contact center metric prediction 172based on the best learning model 155 for the time duration. With thedevelopment of the best learning model 155, the predictor device 116 canutilize the best learning model 155 to assist the server device 112 ofthe contact center 100 with the development of a variety contact centermetrics 172, such as estimated wait times.

In one arrangement, with reference to FIG. 1, following development ofthe best learning model 155, the server device 112 can generate andtransmit a prediction request 170 for an operating condition, such as anestimated wait time (EWT), associated with the contact center 100. Theprediction request 170 can include particular conditions or operatingcriteria 171 which relate to the contact center 100, such as queue size,number of working agents 124, and time frame (e.g., 10 AM to 10 PM), fora given time period (e.g., February).

In response to receipt of the prediction request 170, the predictordevice 116 can extract the operating criteria 171 from the request 170and can apply the operating criteria 171 to the best learning model 155for the time period identified in the request 170 (e.g., February). Byapplying the operating criteria 171 to the best learning model 155 forthe given time period, the predictor device 116 can generate a contactcenter metric prediction 172, such as estimated wait time associatedwith the contact center 100. The predictor device 116 can then forwardthe contact center metric prediction 172 within the contact center 100,such as to the server device 112. The server device 112, in turn, canutilize the contact center metric prediction 172 to determine how toaddress the users 122 and the associated wait times within the contactcenter 100.

As provided above, the predictor device 116 is configured to generate abest learning model 155 for a given time period associated with thecontact center 100. Following generation of the best learning model 155,the predictor device 116 can store the best learning model 155 in avariety of types of databases.

In one arrangement, with reference to FIG. 2, the LOL engine 134 of thepredictor device 116 can include a best learner database 160. In onearrangement, the best learner database 160 can store the data model 150having the largest score, in this case the best learning model 155within the best learner database 160. During operation, as the LOLengine 134 repeats the modelling process for subsequent time periods, aswill be described below, the LOL engine 134 can add to the best learnerdatabase 160 to include best learning models 155 having the largestscore for subsequent time periods (e.g., subsequent months). As such,the predictor device 116 can save the model 155, and associatedalgorithm, with the best scoring function for posterity and futurepredictions.

In one arrangement, the LOL engine 134 of the predictor device 116 caninclude a historical best learner database 162 which is configured tostore the model 155 having the largest overall score for an extendedtime period. For example, as the LOL engine 134 repeats the modellingprocess for subsequent time periods (e.g., for subsequent months), theengine 134 can compare the highest score of the model for eachsubsequent time period with the score of the model stored in thehistorical best learner database 162. In the case where the score of themodel for a successive time period is greater than the score of themodel stored in the historical best database 162, the LOL engine 134 canupdate the historical best learner database 162 to replace thecurrently-stored model with the model 155 from the successive timeperiod having the higher score.

As indicated above, the predictor device 116 can repeat the process ofdetermining a best learning model 155 for subsequent time periods, suchas for subsequent months. For example, with reference to FIG. 1, themonitor service 130 of the predictor device 116 can be configured toretrieve updated contact center operational data 192 from the contactcenter 100 on a monthly basis for subsequent time durations (e.g.,February, March, etc.). With receipt of the updated contact centeroperational data 192, the predictor device 116 can access the algorithmand associated hyperparameter configuration store 138 which includes thealgorithm reference list 139 and hyperparameter options list 141 and cantrain the algorithms and their available hyperparameters provided in thelists 138, 139 with the updated contact center operational data 192. Asa result of the training, the predictor device 116 can generate a set ofsubsequent data models 155-N for the subsequent time duration. Similarto the process described above, following generation of the set ofsubsequent data models 155-N, the predictor device 116 is configured togenerate a score associated with each subsequent data model of the setof subsequent data models which quantifies the performance of eachalgorithm and its available hyperparameters on the contact centeroperational data 192 for the subsequent time duration. The predictordevice 116 can then identify the subsequent data model having thelargest score as the subsequent best learning model for the subsequenttime duration.

Over time, the predictor device 116 can add a best learning model 155 tothe best learner database 160 for each subsequent time duration, such asfor subsequent months. As such, the predictor device 116 can developmodels 155 that account for the changes in the behavior of the contactcenter 100 over the course of a year. For example, the models 155contained in the historical best database 160 can identify the contactcenter 100 as being exceedingly busy in January and February and lessbusy in July and August of a particular year.

As provided above, following identification of a data model having thelargest score as the best learning model 115, the predictor device 116can store the best learning models 155 within a best learner database160. In one arrangement, to account for seasonality, the predictordevice 116 is configured to store two versions of the best learningmodel 155 within the best learner database 160. For example, withreference to FIG. 4, the best learner database 160 includes a bestlearner training portion 180 and a best learner pristine portion 182.

The best learner training portion 180 stores a copy of the best learningmodel 155 which the predictor device 116 can continue to train. Bystoring a copy of the best learning model 155 within the best learnertraining portion 180, the predictor device 116 can maintain a relativelylong, historical trend of previously well-functioning models. As such,seasonality of the best learning model information can be achieved bythe predictor device 116.

The best learner pristine portion 182 stores a copy 156 of the bestlearning model 155 which the predictor device 116 maintains in anunchanged or pristine state after its period of being the best learner.By storing a copy 156 of the best learning model 155 within the bestlearner pristine portion 182, the predictor device 116 allows previouslyutilized best learning model 155 to be re-used at a later time if it hadfunctioned well previously. For example, the copy 156-1 of the bestlearning model 155-1 for the month of December, which is stored withinthe best learner pristine portion 182, will remain unchanged by thepredictor device 116 in light of updated contact center operational data136 received in subsequent months. As such, the best learner pristineportion 182 of the best learner database 160 will contain, and haveavailable, a version 156-1 of the best learning model 155-1 that hasonly been trained during December. As the year progresses, when thefollowing December arrives, the predictor device 116 can access andutilize the copy 156-1 of the best learning model 155-1 for the previousmonth of December.

As provided above, the LOL engine 134 can be configured to execute ahyperparameter and algorithm optimization function 140 to utilize theavailable contact center operational data 136 from the database 132 totrain a set of algorithms and their available hyperparameters. However,since the LOL engine typically develops thousands of models for a giventime period during the training of the algorithms and availablehyperparameters with the contact center operational data 136, operationof the LOL engine 134 can be considered to be relatively slow.Accordingly, the predictor device 116 can be configured to execute theLOL engine 134 periodically during relatively large time intervals(e.g., weekly, monthly, etc.) which can cause updated contact centeroperational data 136 to age and potentially become unusable (e.g.,stale) over those large intervals.

In order to utilize updated contact center operational data 136 duringrelatively shorter time periods, in one arrangement and with referenceto FIG. 5, the predictor device 116 can be configured to execute aPredictor of Predictors (POP) training engine 190. The POP engine 190provides a relatively quicker training approach, relative to the LOLengine 134, which mitigates the contact center operational data 136 fromgoing stale and accounts for short-term fluctuations contact centeroperational data 136 received from the server device 112.

FIG. 5 illustrates a predictor device 116 having an LOL engine 134 and aPOP engine 190. The POP engine 190 is configured to receive updatedcontact center operational data 192 and to train the previous bestlearning models 155, as stored by the predictor device 116 in the bestlearner training portion 180 of the database 160, on the contact centeroperational data 192. Since different algorithms may perform better atdifferent times, the POP engine 190 can weigh the previous best learningmodels 155 to identify the best algorithms to use when generatingcontact center metric predictions 172 (e.g., estimated wait times) forthe contact center 100.

During operation, the predictor device 116 receives the updated contactcenter operational data 192 from the contact center 100 for a timeduration. For example, the predictor device 116 provides API callinstructions 139 which directs the monitor service 130 to retrieveupdated contact center operational data 192 from the server device 112.In the case where the predictor device 116 detects the transmission of aprevious API call to the LOL engine 134, the predictor device 116further provides an API call 143 to the POP engine 190. This API call143 causes the POP engine 190 to access the best learner trainingportion 180 of the best learner database 160.

After accessing the best learner training portion 180, the POP engine190 is configured to partially train the current or most-recent bestlearning model 155 stored within the best learner database 160 on theupdated contact center operational data 192 and to partially train eachof the best learning models previously stored within the best learnerdatabase 160 on the updated contact center operational data 192. Forexample, in the case where the best learner database 160 includes a bestlearning model 155-3 for the month of February, a best learning model155-2 for the month of January, and best learning model 155-1 for themonth of December, the POP engine 190 can use the updated contact centeroperational data 192 to train the most recent best learning model 155-3(February) and the previous best learning models 155-2, 155-1 (Januaryand December). In the process of partial training, the POP engine 190can access each previously constructed model 155-1 through 155-3 alongwith and the pre-trained weights for each model 155-1 through 155-3. ThePOP engine 190 can then hold the pre-trained weights constant up to thefinal layer of each of the models 155-1 through 155-3. As the POP engine190 trains the models 155 155-1 through 155-3 on the updated contactcenter operational data 192, the POP engine 190 allows the weights infinal layer to change. As a result of the partial training, the POPengine 190 can generate trained best learning models 158-1, 158-2, and158-3, respectively.

Following the partial training, the POP engine 190 can weigh the trainedbest learning model 158-1 through 158-3 for each month with the pristinemodel 156-1 through 156-3 for that month, as stored in the best learnerpristine portion 182 of the database 160 (i.e., with a minimum weightinggiven to the current best predictor) to generate an associated accuracyscore 160 for each. For example, based on the weighing process, the POPengine 190 can generate various accuracy scores, such as a 98% accuracyscore 160-3 for the best learning model 158-3 of February, a 90%accuracy score 160-2 for the trained best learning model 158-2 ofJanuary, and a 95% accuracy score 160-1 for the trained best learningmodel 158-1 of December.

Further, in response to receiving contact center operation criteria 171from the server device 112, such as queue size, number of working agents124, and time frame (e.g., 10 AM to 10 PM), for a given time period(e.g., February), the predictor device 116 can apply the trained bestlearning model 158 having the highest accuracy score to the contactcenter operation criteria 171 to generate a contact center metricprediction 172 for the contact center 100. Using the above scores as anexample, the POP engine 190 can identify the trained best learning model158-3 of February as having the highest accuracy score and, as such, theappropriate model to use when generating the prediction 172 (e.g.,estimated wait time) for the contact center 100 at that time.

As provided above, following the generation of a set of data models 150,the predictor device 116 is configured to generate a score 152 whichquantifies the accuracy or quality for each model 150 during training.While the predictor device 116 can generate the scores 152 in a varietyof ways, in one arrangement and with reference to FIG. 6, the predictordevice 116 is configured to utilize at least one training quality metric250 to generate the score 152 and to allow assessment of the accuracy ofeach model 150.

For example, following the generation of each model 150, the predictordevice 116 is configured to apply the training quality metric 250 to thecontact center operational data 136 and to each data model 150. In onearrangement, the training quality metric 250 can be configured as afunction which quantifies the fit between a model 150 and a particulardata set to the model 150. As such, by applying the training qualitymetric 250 to the contact center operational data 136 and to each datamodel 150, the predictor device 116 can identify the fit and cangenerate, as an output a score or model quality value 152 associatedwith each model 150.

The predictor device 116 can be configured to utilize a variety of typesof training quality metric 250 to generate the model quality value 152for each model 150 during the training process. The following providesseveral examples of the training quality metrics 250 utilized by thepredictor device 116 during operation.

In one arrangement, during the training process, the predictor device116 is configured to utilize a mean absolute error (MAE) metric 252 asthe training quality metric 250. Mean absolute error relates to ameasured difference between two variables. As such, during operation,the predictor device 116 can utilize each model 150 to identify thepredicted output for a particular variable, such as EWT, and can utilizethe contact center operational data 136 to identify the actual outputvalue for a particular variable, such as EWT. The predictor device 116can then apply the following MAE metric 252 to both the model 150 andthe contact center operational data 136:

${MAE} = {\frac{1}{n}{\sum{{{y - \hat{y}}}.}}}$

For each actual output value, y, of the contact center operational data136, the predictor device 116 utilizes the MAE metric 252 to identify amagnitude of a residual, y−y{circumflex over ( )}, where andy{circumflex over ( )} is the predicted output value from the model 150.The MAE metric 252 utilizes the absolute value of the residual tomitigate the cancellation of negative and positive residual values. Thepredictor device 116 further utilizes the MAE metric 252 to calculatethe average of the residual values, where n is the total number of datapoints within the contact center operational data 136. The predictordevice 116 provides the average of the residual values as a mean errorscore 253 the given model 150. The predictor device 116 can output themean error score 253 as the model quality value 152 for the model 150.

It is noted that each residual value contributes proportionally to thetotal amount of error calculated by the MAE metric 252 which means thatlarger errors will contribute linearly to the overall error. As aresult, a relatively smaller model quality value 152 resulting fromapplication of the MAE metric 252 suggests that a model 150 has arelatively high predictive accuracy. By contrast, a relatively largermodel quality value 152 resulting from application of the MAE metric 252suggests that the model 150 may have a lower predictive accuracy.

In one arrangement, during the training process, the predictor device116 is configured to utilize an explained variance or variation (EV)metric 254 as the training quality metric 250. Generally, EV identifiesthe level to which a model can account for variation found within a setof data. In other words, the EV is indicative of a model's totalvariance that can explained by factors that are actually present andthat are not caused by error variance.

With application of the EV metric 254 to the contact center operationaldata 136 and to each model 150, the predictor device 116 is configuredto identify any discrepancy between the model 150 and the actual contactcenter operational data 136. For example, during application of the EVmetric 254, the predictor device 116 can identify a coefficient ofdetermination for the contact center operational data 136 relative to agiven model 150. The predictor device 116 provides coefficient ofdetermination as an explained variance score 255 for the given model150. The predictor device 116 can output the explained variance score255 as the model quality value 152 for the model 150.

The coefficient of determination indicates of the number of dataelements of the contact center operational data 136 which fall within aregression line defined by a particular model 150. As such, therelatively higher the explained variance score 255, the greater theaccuracy of the model 150. For example, a score of 1.0, while typicallyunlikely, identifies a 100% accuracy of the model 150 while a relativelower values suggests a lower level of accuracy of the model 150 (e.g.,EV>=0.60 indicates that the model 150 is >60% accurate).

In one arrangement, during the training process, the predictor device116 is configured to utilize a cross-validation metric 256 as thetraining quality metric 250. Generally, cross-validation relates to atechnique which evaluates a predictive model by partitioning an originalsample into a training set to train the model, and a test set to testthe model. While the predictor device 116 can utilize a variety ofcross-validation metric 256, in one arrangement, the predictor device116 is configured to utilize a k-fold cross-validation metric 256, asdescribed below.

With application of the k-fold cross-validation metric 256, thepredictor device 116 randomly partitions the contact center operationaldata 136 into k equal size sub-samples. For example the predictor device116 can divide the contact center operational data 136 into five equalsubsets. Of these 5 subsets (k), the predictor device 116 retains asingle subset as a validation data set to test each model 150. Thepredictor device 116 further retains each of the remaining 4 (k−1)subsets to train the model 150. With such partitioning, the predictordevice 116 uses all of the observations within the contact centeroperational data 136 for both training and validation, and utilizes eachobservation for validation exactly once.

During operation, the predictor device 116 repeats the application ofthe cross-validation metric 256 to the model 150 k times (e.g., thefolds), with each of the k subsets used exactly once as the validationdata relative to the model 150. The predictor device 116 can thenaverage or otherwise combine the k results to generate a singlevalidation score 257 for the model 150. The predictor device 116 canoutput the validation score 257 as the model quality value 152 for themodel 150.

As provided above, the predictor device 116 is configured to quantifythe accuracy or quality for each model 150 of a set of models duringtraining. As such, during the training process, the predictor device 116can select a model 150 having an indication of the relatively highestquality, as evidenced by a score 152, as the best learning model 155 andcan deploy that selected best learning model 155 to generate a contactcenter metric prediction 172, such as EWT. In one arrangement, thepredictor device 116 is configured to monitor the quality of the bestlearner model 155 which has been deployed following training and isbeing used online.

FIG. 7 illustrates a schematic representation of the predictor device116 which is configured to evaluate the quality of a deployed bestlearning model 155 over time. In one arrangement, the predictor device116 is configured to monitor performance and fitting (e.g., the quality)of the best learning model 155 by evaluating subsequently receivedtraining data (e.g., updated contact center operational data 192)against the best learning model 155.

For example, assume the case where the predictor device 116 has deployeda best learning model 155 having a quality score 152 of 90% to generatecontact center metric predictions 172. As provided above, the predictordevice 116 can also include a database 132 which stores contact centeroperational data 136 and updated contact center operational data 192 asretrieved by a monitor service 130 over time. With such a configuration,the predictor device 116 is configured to periodically evaluate thequality of the deployed best learning model 155 (i.e., the quality score152) over time utilizing operational data stored by the database 132,such as the updated contact center operational data 192.

During the quality evaluation process, the predictor device 116 isconfigured to apply at least one on-line quality metric 300 to theupdated contact center operational data 192 and to the best learningmodel 155 to generate a quality score 302 for the best learning model155. While the on-line quality metric 300 can be configured in a varietyof ways, in one arrangement, the on-line quality metric 300 can relateto the difference between a mean of the updated contact centeroperational data 192 (MoGT) and a mean of the data generated by the bestlearning model 155 (MoP) and can be given as the relationship|MoP−MoGT|. For example, in the case where MoP=5.601 and MoGT=5.598, thequality score 302 for the mean=0.003. In one arrangement, the on-linequality metric 300 can relate to the difference between the standarddeviation of the updated contact center operational data 192 (StdGT) andthe standard deviation of the data generated by the best learning model155 (StdP) and can be given as the relationship |StdP−StdGT|. Forexample, in the case where StdP=20.817 and StdGT=18.262, the qualityscore 302 for the standard deviation=2.555. In one arrangement, theon-line quality metric 300 can relate to the difference between amaximum value (MaxGT) of the updated contact center operational data 192(MaxGT) and a maximum value (MaxP) of the of the data generated by thebest learning model 155 and can be given as the relationship|MaxP−MaxGT|. For example, in the case where MaxP=579.0 and MaxGT=612.3,the quality score 302 for the maximum value=33.3.

The predictor device 116 is then configured to compare the quality score302 with a quality threshold 304. While the quality threshold 304 can beconfigured in a variety of ways, in one arrangement, the qualitythreshold 304 can relate to a ratio of the standard deviation of theupdated contact center operational data 192 (StdGT) and mean of theupdated contact center operational data 192 (MoGT) and can be given asthe relationship (StdGT/MoGT). In one arrangement, the quality threshold304 can relate to a multiple of the standard deviation of the updatedcontact center operational data 192 (StdGT) and can be given as therelationship 3 * StdGT. For the comparison, the predictor device 116 canbe configured to compare the quality score 302 with the qualitythreshold 304 as follows:|MoP−MoGT|<=(StdGT/MoGT)|StdP−StdGT|<=(StdGT/MoGT)|MaxP−MaxGT|<=3*StdGT.

In the case where the predictor device 116 identifies the quality score302 as meeting the relationship with the quality threshold 304, asprovided above, such identification indicates that the performance andfitting (e.g., the quality) of the best learning model 155 is adequateand no further action is necessary during this evaluation period. In thecase where the predictor device 116 identifies the quality score 302 asfalling outside of the quality threshold 304 (e.g., as failing to meetthe relationship with the quality threshold 304), the predictor device116 is configured to execute a quality correction function 306.

The quality correction function 306 can be configured in a variety ofways. For example, in the case where the predictor device 116 identifiesthe best learning model 155 as deviating from the subsequently receivedtraining data (e.g., updated contact center operational data 192), thepredictor device 116 can execute the quality correction function 306 tocauses a retraining and re-evaluation of the best learning model 155with updated contact center operational data 192. In another example,execution of the quality correction function 306 can cause the predictordevice 116 to provide a notification, such as an email notification, toa contact center operator identifying the discrepancy. In anotherexample, execution of the quality correction function 306 can cause thepredictor device 116 to select another model from the set of models 150identified in FIG. 2 (e.g., a model 150 having the next largest score152) and to deploy the newly selected model as the best learning model155.

While various embodiments of the innovation have been particularly shownand described, it will be understood by those skilled in the art thatvarious changes in form and details may be made therein withoutdeparting from the spirit and scope of the innovation as defined by theappended claims.

What is claimed is:
 1. In a predictor device, a method for predicting ametric of a contact center, comprising: receiving, by the predictordevice, contact center operational data associated with a time duration;training, by the predictor device, a set of algorithms and theiravailable hyperparameters with the contact center operational data togenerate a set of data models; generating, by the predictor device, ascore associated with each data model of the set of data models, thescore quantifying a performance of each algorithm and its availablehyperparameters on the contact center operational data; identifying, bythe predictor device, the data model having the largest score as a bestlearning model for the time duration; and generating, by the predictordevice, a contact center metric prediction based on the best learningmodel for the time duration.
 2. The method of claim 1, whereingenerating the contact center metric prediction based on the bestlearning model for the time duration comprises: receiving, by thepredictor device, a prediction request, the prediction request includingoperating criteria which relate to the contact center; applying, by thepredictor device, the operating criteria to the best learning model forthe time duration to generate the contact center metric prediction; andforwarding, by the predictor device, the contact center metricprediction to the contact center.
 3. The method of claim 1, furthercomprising repeating the steps of: receiving, by the predictor device,contact center operational data for a subsequent time duration;training, by the predictor device, the set of algorithms and theiravailable hyperparameters with the contact center operational data forthe subsequent time duration to generate a set of subsequent datamodels; generating, by the predictor device, a score associated witheach subsequent data model of the set of subsequent data models, thescore quantifying the performance of each algorithm and its availablehyperparameters on the contact center operational data for thesubsequent time duration; and identifying, by the predictor device, thesubsequent data model having the largest score as the subsequent bestlearning model for the subsequent time duration.
 4. The method of claim1, wherein identifying the data model having the largest score as thebest learning model, further comprises: storing, by the predictordevice, the best learning model as a pristine best learning model forthe time duration in a best learner pristine portion of a best learnerdatabase; and storing, by the predictor device, the best learning modelas a training best learning model for the time duration in a bestlearner training portion of the best learner database.
 5. The method ofclaim 4, comprising: receiving, by the predictor device, updated contactcenter operational data associated with the time duration; accessing, bythe predictor device, the best learner training portion of the bestlearner database; partially training, by the predictor device, amost-recent best learning model stored within the best learner trainingportion of the best learner database on the updated contact centeroperational data; partially training, by the predictor device, at leastone subsequent best learning model stored within the best learnertraining portion of the best learner database on the updated contactcenter operational data; weighing, by the predictor device, each trainedbest learning model with a corresponding pristine best learning modelstored within the best learner pristine portion of the best learnerdatabase to generate the score for each trained best learning model. 6.The method of claim 5, comprising, in response to receiving contactcenter operation criteria, applying, by the predictor device, thetrained best learning model having the highest score to the contactcenter operation criteria to generate the contact center metricprediction for the contact center.
 7. The method of claim 1, whereingenerating the score associated with each data model of the set of datamodels comprises: applying, by the predictor device, at least onetraining quality metric to the contact center operational data and toeach data model of the set of data models; and generating, by thepredictor device, a model quality value for each data model of the setof data models based upon application of the at least one trainingquality metric.
 8. The method of claim 7, wherein: applying the at leastone training quality metric to the contact center operational data andto each data model of the set of data models comprises applying, by thepredictor device, a mean absolute error metric to the contact centeroperational data and to each data model of the set of data models; andgenerating the model quality value for each data model of the set ofdata models comprises generating, by the predictor device, a mean errorscore for each data model of the set of data models.
 9. The method ofclaim 7, wherein: applying the at least one training quality metric tothe contact center operational data and to each data model of the set ofdata models comprises applying, by the predictor device, an explainedvariance metric to the contact center operational data and to each datamodel of the set of data models; and generating the model quality valuefor each data model of the set of data models comprises generating, bythe predictor device, an explained variance score for each data model ofthe set of data models.
 10. The method of claim 7, wherein: applying theat least one training quality metric to the contact center operationaldata and to each data model of the set of data models comprisesapplying, by the predictor device, a cross-validation metric to thecontact center operational data and to each data model of the set ofdata models; and generating the model quality value for each data modelof the set of data models comprises generating, by the predictor device,a validation score for each data model of the set of data models. 11.The method of claim 1, further comprising: applying, by the predictordevice, at least one on-line quality metric to the contact centeroperational data and to the best learning model to generate a qualityscore for the best learning model; and when the quality score of thebest learning model for the time duration falls outside of a qualitythreshold, executing, by the predictor device, a quality correctionfunction.
 12. A predictor device configured to predict a metric of acontact center, the predictor device comprising: a controller having aprocessor and a memory, the controller configured to: receive contactcenter operational data associated with a time duration; train a set ofalgorithms and their available hyperparameters with the contact centeroperational data to generate a set of data models; generate a scoreassociated with each data model of the set of data models, the scorequantifying a performance of each algorithm and its availablehyperparameters on the contact center operational data; identify thedata model having the largest score as a best learning model for thetime duration; and generate a contact center metric prediction based onthe best learning model for the time duration.
 13. The predictor deviceof claim 12, wherein when generating the contact center metricprediction based on the best learning model for the time duration thecontroller is configured to: receive a prediction request, theprediction request including operating criteria which relate to thecontact center; apply the operating criteria to the best learning modelfor the time duration to generate the contact center metric prediction;and forward the contact center metric prediction to the contact center.14. The predictor device of claim 12, wherein the controller isconfigured to further: receive contact center operational data for asubsequent time duration; train the set of algorithms and theiravailable hyperparameters with the contact center operational data forthe subsequent time duration to generate a set of subsequent datamodels; generate a score associated with each subsequent data model ofthe set of subsequent data models, the score quantifying the performanceof each algorithm and its available hyperparameters on the contactcenter operational data for the subsequent time duration; and identifythe subsequent data model having the largest score as the subsequentbest learning model for the subsequent time duration.
 15. The predictordevice of claim 12, wherein when identifying the data model having thelargest score as the best learning model, the controller is furtherconfigured to: store the best learning model as a pristine best learningmodel for the time duration in a best learner pristine portion of a bestlearner database; and store the best learning model as a training bestlearning model for the time duration in a best learner training portionof the best learner database.
 16. The predictor device of claim 15,wherein the controller is configured to: receive updated contact centeroperational data associated with the time duration; access the bestlearner training portion of the best learner database; partially train amost-recent best learning model stored within the best learner trainingportion of the best learner database on the updated contact centeroperational data; partially train at least one subsequent best learningmodel stored within the best learner training portion of the bestlearner database on the updated contact center operational data; weigheach trained best learning model with a corresponding pristine bestlearning model stored within the best learner pristine portion of thebest learner database to generate the score for each trained bestlearning model.
 17. The predictor device of claim 16, wherein, inresponse to receiving contact center operation criteria, the controlleris configured to apply the trained best learning model having thehighest score to the contact center operation criteria to generate thecontact center metric prediction for the contact center.
 18. Thepredictor device of claim 12, wherein when generating the scoreassociated with each data model of the set of data models, thecontroller is configured to: apply at least one training quality metricto the contact center operational data and to each data model of the setof data models; and generate a model quality value for each data modelof the set of data models based upon application of the at least onetraining quality metric.
 19. The predictor device of claim 18, wherein:when applying the at least one training quality metric to the contactcenter operational data and to each data model of the set of datamodels, the controller is configured to apply a mean absolute errormetric to the contact center operational data and to each data model ofthe set of data models; and when generating the model quality value foreach data model of the set of data models, the controller is configuredto generate a mean error score for each data model of the set of datamodels.
 20. The predictor device of claim 18, wherein: when applying theat least one training quality metric to the contact center operationaldata and to each data model of the set of data models the controller isconfigured to apply an explained variance metric to the contact centeroperational data and to each data model of the set of data models; andwhen generating the model quality value for each data model of the setof data models the controller is configured to generate an explainedvariance score for each data model of the set of data models.
 21. Thepredictor device of claim 18, wherein: when applying the at least onetraining quality metric to the contact center operational data and toeach data model of the set of data models the controller is configuredto apply a cross-validation metric to the contact center operationaldata and to each data model of the set of data models; and whengenerating the model quality value for each data model of the set ofdata models the controller is configured to generate a validation scorefor each data model of the set of data models.
 22. The predictor deviceof claim 13, wherein the controller is further configured to: apply atleast one on-line quality metric to the contact center operational dataand to the best learning model to generate a quality score for the bestlearning model; and when the quality score of the best learning modelfor the time duration falls outside of a quality threshold, execute aquality correction function.